systematic comparison
A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models
Eisape, Tiwalayo, Tessler, MH, Dasgupta, Ishita, Sha, Fei, van Steenkiste, Sjoerd, Linzen, Tal
A central component of rational behavior is logical inference: the process of determining which conclusions follow from a set of premises. Psychologists have documented several ways in which humans' inferences deviate from the rules of logic. Do language models, which are trained on text generated by humans, replicate these biases, or are they able to overcome them? Focusing on the case of syllogisms -- inferences from two simple premises, which have been studied extensively in psychology -- we show that larger models are more logical than smaller ones, and also more logical than humans. At the same time, even the largest models make systematic errors, some of which mirror human reasoning biases such as ordering effects and logical fallacies. Overall, we find that language models mimic the human biases included in their training data, but are able to overcome them in some cases.
Systematic Comparison of Software Agents and Digital Twins: Differences, Similarities, and Synergies in Industrial Production
Reinpold, Lasse Matthias, Wagner, Lukas Peter, Gehlhoff, Felix, Ramonat, Malte, Kilthau, Maximilian, Gill, Milapji Singh, Reif, Jonathan Tobias, Henkel, Vincent, Scholz, Lena, Fay, Alexander
To achieve a highly agile and flexible production, it is envisioned that industrial production systems gradually become more decentralized, interconnected, and intelligent. Within this vision, production assets collaborate with each other, exhibiting a high degree of autonomy. Furthermore, knowledge about individual production assets is readily available throughout their entire life-cycles. To realize this vision, adequate use of information technology is required. Two commonly applied software paradigms in this context are Software Agents (referred to as Agents) and Digital Twins (DTs). This work presents a systematic comparison of Agents and DTs in industrial applications. The goal of the study is to determine the differences, similarities, and potential synergies between the two paradigms. The comparison is based on the purposes for which Agents and DTs are applied, the properties and capabilities exhibited by these software paradigms, and how they can be allocated within the Reference Architecture Model Industry 4.0. The comparison reveals that Agents are commonly employed in the collaborative planning and execution of production processes, while DTs typically play a more passive role in monitoring production resources and processing information. Although these observations imply characteristic sets of capabilities and properties for both Agents and DTs, a clear and definitive distinction between the two paradigms cannot be made. Instead, the analysis indicates that production assets utilizing a combination of Agents and DTs would demonstrate high degrees of intelligence, autonomy, sociability, and fidelity. To achieve this, further standardization is required, particularly in the field of DTs.
Using deep neural networks to predict how natural sounds are processed by the brain
In recent years, machine learning techniques have accelerated and innovated research in numerous fields, including neuroscience. By identifying patterns in experimental data, these models could for instance predict the neural processes associated with specific experiences or with the processing of sensory stimuli. Researchers at CNRS and Universitรฉ Aix-Marseille and Maastricht University recently tried to use computational models to predict how the human brain transforms sounds into semantic representations of what is happening in the surrounding environment. Their paper, published in Nature Neuroscience, shows that some deep neural network (DNN)-based models might be better at predicting neural processes from neuroimaging and experimental data. "Our main interest is to make numerical predictions about how natural sounds are perceived and represented in the brain, and to use computational models to understand how we transform the heard acoustic signal into a semantic representation of the objects and events in the auditory environment," Bruno Giordano, one of the researchers who carried out the study, told Medical Xpress.
Dynamic Features for Visual Speechreading: A Systematic Comparison
Humans use visual as well as auditory speech signals to recognize spoken words. A variety of systems have been investigated for per(cid:173) forming this task. The main purpose of this research was to sys(cid:173) tematically compare the performance of a range of dynamic visual features on a speechreading task. We have found that normal(cid:173) ization of images to eliminate variation due to translation, scale, and planar rotation yielded substantial improvements in general(cid:173) ization performance regardless of the visual representation used. In addition, the dynamic information in the difference between suc(cid:173) cessive frames yielded better performance than optical-flow based approaches, and compression by local low-pass filtering worked sur(cid:173) prisingly better than global principal components analysis (PCA).
Brain-age prediction: a systematic comparison of machine learning workflows
The difference between age predicted using anatomical brain scans and chronological age, i.e., the brain-age delta, provides a proxy for atypical aging. Various data representations and machine learning (ML) algorithms have been used for brain-age estimation. However, how these choices compare on performance criteria important for real-world applications, such as; (1) within-site accuracy, (2) cross-site generalization, (3) test-retest reliability, and (4) longitudinal consistency, remains uncharacterized. We evaluated 128 workflows consisting of 16 feature representations derived from gray matter (GM) images and eight ML algorithms with diverse inductive biases. Using four large neuroimaging databases covering the adult lifespan (total N 2953, 18-88 years), we followed a systematic model selection procedure by sequentially applying stringent criteria. The 128 workflows showed a within-site mean absolute error (MAE) between 4.73-8.38
Metrics and methods for a systematic comparison of fairness-aware machine learning algorithms
Jones, Gareth P., Hickey, James M., Di Stefano, Pietro G., Dhanjal, Charanpal, Stoddart, Laura C., Vasileiou, Vlasios
Understanding and removing bias from the decisions made by machine learning models is essential to avoid discrimination against unprivileged groups. Despite recent progress in algorithmic fairness, there is still no clear answer as to which bias-mitigation approaches are most effective. Evaluation strategies are typically use-case specific, rely on data with unclear bias, and employ a fixed policy to convert model outputs to decision outcomes. To address these problems, we performed a systematic comparison of a number of popular fairness algorithms applicable to supervised classification. Our study is the most comprehensive of its kind. It utilizes three real and four synthetic datasets, and two different ways of converting model outputs to decisions. It considers fairness, predictive-performance, calibration quality, and speed of 28 different modelling pipelines, corresponding to both fairness-unaware and fairness-aware algorithms. We found that fairness-unaware algorithms typically fail to produce adequately fair models and that the simplest algorithms are not necessarily the fairest ones. We also found that fairness-aware algorithms can induce fairness without material drops in predictive power. Finally, we found that dataset idiosyncracies (e.g., degree of intrinsic unfairness, nature of correlations) do affect the performance of fairness-aware approaches. Our results allow the practitioner to narrow down the approach(es) they would like to adopt without having to know in advance their fairness requirements.
Dynamic Features for Visual Speechreading: A Systematic Comparison
Gray, Michael S., Movellan, Javier R., Sejnowski, Terrence J.
Humans use visual as well as auditory speech signals to recognize spoken words. A variety of systems have been investigated for performing this task. The main purpose of this research was to systematically compare the performance of a range of dynamic visual features on a speechreading task. We have found that normalization of images to eliminate variation due to translation, scale, and planar rotation yielded substantial improvements in generalization performance regardless of the visual representation used. In addition, the dynamic information in the difference between successive frames yielded better performance than optical-flow based approaches, and compression by local low-pass filtering worked surprisingly better than global principal components analysis (PCA). These results are examined and possible explanations are explored.
Dynamic Features for Visual Speechreading: A Systematic Comparison
Gray, Michael S., Movellan, Javier R., Sejnowski, Terrence J.
Humans use visual as well as auditory speech signals to recognize spoken words. A variety of systems have been investigated for performing this task. The main purpose of this research was to systematically compare the performance of a range of dynamic visual features on a speechreading task. We have found that normalization of images to eliminate variation due to translation, scale, and planar rotation yielded substantial improvements in generalization performance regardless of the visual representation used. In addition, the dynamic information in the difference between successive frames yielded better performance than optical-flow based approaches, and compression by local low-pass filtering worked surprisingly better than global principal components analysis (PCA). These results are examined and possible explanations are explored.
Dynamic Features for Visual Speechreading: A Systematic Comparison
Gray, Michael S., Movellan, Javier R., Sejnowski, Terrence J.
Humans use visual as well as auditory speech signals to recognize spoken words. A variety of systems have been investigated for performing thistask. The main purpose of this research was to systematically comparethe performance of a range of dynamic visual features on a speechreading task. We have found that normalization ofimages to eliminate variation due to translation, scale, and planar rotation yielded substantial improvements in generalization performanceregardless of the visual representation used. In addition, the dynamic information in the difference between successive framesyielded better performance than optical-flow based approaches, and compression by local low-pass filtering worked surprisingly betterthan global principal components analysis (PCA). These results are examined and possible explanations are explored.